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I.  INTRODUCTION 


A.  BACKGROUND 

The  Department  of  Defense  (DoD)  fiscal  year  2009  budget  for  Research, 
Development,  Test  and  Evaluation  (RDT&E)  and  procurement  exceeds  $180  billion 
(Gates,  2009,  p.  37).  With  such  huge  budget  outlays  and  the  increasing  pressures  of 
shrinking  discretionary  budgets  in  a  fragile  economy,  the  DoD  Acquisition  System  is  the 
subject  of  intense  scrutiny  from  government  oversight  activities,  industry,  and  the  general 
public.  This  scrutiny  has  been  amplified  by  highly  publicized  acquisition  program 
failures,  continued  cost  and  schedule  overruns,  and  lengthy  development  cycles. 

DoD  acquisition  has  endured  an  environment  of  seemingly  perpetual  reform  to 
arrest  this  chronically  poor  performance,  resulting  in  complex  acquisition  process 
models,  increased  executive  oversight,  and  incremental  policy  changes.  Continued  reform 
is  certain,  as  Defense  Secretary  Gates  repeatedly  expresses  frustration  with  the 
acquisition  process.  He  cites  examples  that  reflect  the  need  for  change:  a  need  to  conduct 
the  recent  acquisitions  of  the  Mine  Resistant  Ambush  Protected  vehicles  (MRAP)  and 
Intelligence  Surveillance  Reconnaissance  capabilities  outside  normal  acquisition 
processes.  Secretary  Gates  has  also  called  for  a  change  in  the  mindset  of  those  in  the 
acquisition  community  to  accept  75%  solutions,  vice  the  99%  solutions  often  overreached 
by  typical  acquisition  programs  (pp.  37-38). 

The  effectiveness  of  these  reforms  is  not  yet  evidenced  in  the  overall  performance 
of  the  DoD  Acquisition  System.  Independent  and  government-chartered  studies  and 
reports  have  repeatedly  highlighted  the  need  for  improved  systems  engineering  and 
business  processes  to  incorporate  best  practices  from  the  commercial  sector.  In  the 
Government  Accounting  Office  (GAO)  FY08  review  of  select  DoD  Acquisition 
Programs,  the  GAO  found  that  total  acquisition  costs  increased  26%,  and  development 
costs  increased  by  40%  from  first  estimates,  with  program  schedule  delays  averaging  2 1 
months.  The  GAO  also  noted  the  “continuing  absence  of  knowledge-based  acquisition 


1 


processes  steeped  in  disciplined  systems  engineering  practices — aimed  at  analyzing 
requirements  to  determine  their  reasonableness  before  a  program  starts — contributed 
significantly  to  this”  (GAO,  2008,  March,  p.  5). 

The  DoD  embraced  several  recommendations  from  these  critical  reports  and 
moved  to  adopt  several  commercial  best  practices  and  process  initiatives.  Two  such 
policy  changes  relevant  to  this  research  are  the  adoption  of  Capability  Portfolio 
Management  (CPM)  and  Open  Architecture  (OA)  approaches,  discussed  at  length  in 
Chapter  II.  CPM  and  OA  are  relatively  early  in  their  implementation  and  address 
different  levels  of  the  acquisition  process,  but  they  reflect  the  overarching  DoD  goals  of 
improving  decision  making  regarding  systems-of-systems  (SoS)  acquisitions  to  avoid 
duplication,  identify  gaps,  and  decrease  costs  and  development  times. 

The  tools  and  processes  used  by  acquisition  decision  makers  to  implement  CPM 
and  OA  policies  are  not  well  defined.  A  fundamental  requirement  of  both  CPM  and  OA 
approaches  is  that  acquisition  managers  develop  an  awareness  of  related  efforts  and 
activities  across  an  enterprise  and/or  community  of  interest  (COI)  to  support  decision 
making  regarding  duplication  of  effort,  capability  gaps,  re-use  and  collaboration 
opportunities.  It  is  the  premise  of  this  paper  that  development  of  Program  Self-awareness 
is  fundamental  to  the  success  of  the  CPM/OA  reform  initiatives.  This  paper  applies 
commercial  and  government  best  practices  to  develop  Program  Self-awareness  through 
Knowledge  Management  (KM)  methods  and  tools. 

The  DoD  Maritime  Domain  Awareness  (MDA)  Program  will  be  used  as  a  test 
case  for  application  of  KM  decision-support  tools,  providing  relational  views  of  program 
elements  and  attributes — termed  “features” — to  support  informed  program  decision 
making.  This  thesis  asserts  that  application  of  KM  tools  will  improve  Program  Self- 
awareness  and  support  better  decision  making,  which  is  required  to  realize  the  full 
potential  of  CPM  and  OA  initiatives. 

B.  PROBLEM  STATEMENT  AND  RESEARCH  QUESTION 

DoD  acquisition  is  comprised  of  numerous  stakeholders  and  organizations  that 

navigate  procurement  processes  in  an  uncertain  environment  in  order  to  deliver  useful 
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military  capability  to  the  warfighter  at  the  best  possible  value  to  the  government. 
Acquisition  reforms  have  been  largely  ineffective  at  improving  the  efficiency  of  the 
system,  due,  in  part,  to  stovepipe  activities  that  often  result  in  duplication  of  effort,  lack 
of  re-use  and  collaboration  on  related  development  efforts.  It  is  the  goal  of  this  thesis  to 
demonstrate  the  Program  Self-awareness  concept  through  application  of  KM  tools  to  the 
DoD  MDA  Program  to  answer  the  following  research  question. 

•  How  can  Knowledge  Management  methodologies  and  decision-support 
tools  be  used  to  improve  Program  Self-awareness  and  decision  making 
that  will  enable  collaboration  and  re-use  in  complex  DoD  acquisition 
programs? 

C.  METHODOLOGY 

The  Stanford  University  Center  for  Integrated  Facility  Engineering  (CIFE) 
“Horseshoe”  methodology  (Figure  1)  was  used  to  guide  this  research  (Ho,  2007,  p.  2). 
This  research  will  explore  the  problem  of  duplication,  lack  of  re-use,  and  collaboration  in 
DoD  Acquisition  and  follow  the  intuition  that  increased  Program  Self-awareness — 
enabled  by  KM  decision-support  tools — will  improve  acquisition  process  efficiencies  in 
these  areas.  The  research  will  be  grounded  in  Systems  Theory  and  the  Congruence 
Model  to  develop  an  understanding  of  the  DoD  Acquisition  System  and  to  identify  root 
causes  of  the  stated  problem.  The  research  will  then  apply  KM  tools  to  the  DoD  MDA 
Program  as  a  test  case  and  evaluate  the  potential  benefit  of  these  prototype  KM  tools  to 
program  decision  makers.  This  work  will  provide  the  foundation  for  future  research  on 
the  Program  Self-awareness  concept  and  development  of  KM  tools  with  the  goal  of 
improving  decision  making  and  enabling  re-use  and  collaboration  in  DoD  acquisition 
programs. 
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■*•••*■  Compare 


Figure  1.  CIFE  Research  Methodology  (From  Ho,  2007,  p.  2) 


D.  SCOPE 

This  thesis  will  develop  the  foundation  of  the  Program  Self-awareness  concept  to 
support  improved  decision  making,  collaboration,  and  re-use  in  DoD  Acquisition.  It  will 
apply  Systems  Theory  and  Knowledge  Management  principles  and  tools  developed 
during  the  academic  and  technology  review,  grounding  the  Program  Self-awareness 
concept  in  mature  academic  concepts  and  methodologies. 

The  implementation  impact  of  this  research  on  other  organizational  components 
within  the  DoD  Acquisition  System  (structure,  processes,  people)  are  not  addressed  in 
depth  in  this  research.  Further  research  will  be  required  to  study  organizational 
congruence  and  cultural  issues  in  order  to  realize  the  full  benefits  of  the  Program  Self- 
awareness  concept. 
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E. 


ORGANIZATION  OF  THESIS 


Chapter  II  will  build  the  academic  and  technology  foundation  for  the  Program 
Self-Awareness  concept  through  a  review  of  Systems  Theory  and  the  Congruence  Model, 
the  field  of  Knowledge  Management,  and  trends  in  the  DoD  Acquisition  environment 
that  lend  themselves  to  application  of  KM  tools.  Chapter  III  will  introduce  the  concept  of 
Program  Self-awareness  and  apply  the  Congruence  Model  to  describe  the  DoD 
Acquisition  System.  Chapter  IV  will  apply  KM  tools  and  methodologies  to  the  DoD 
MDA  Program  to  identify  feature  clusters  of  select  MDA  technologies  to  demonstrate  the 
potential  for  improved  Program  Self-awareness.  Chapter  V  will  provide  conclusions  and 
recommendations  for  future  research. 
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II.  LITERATURE  AND  TECHNOLOGY  REVIEW 


A.  SYSTEMS  THEORY  AND  ORGANIZATIONS 

This  research  explores  the  potential  for  change  in  the  DoD  Acquisition  System 
through  application  of  KM  tools — resulting  in  improved  Program  Self-awareness.  This 
section  reviews  Systems  Theory  and  the  Congruence  Model  to  provide  a  framework  to 
understand  the  complexity  of  the  DoD  Acquisition  system  described  in  later  parts  of  this 
research. 

1.  The  Leavitt  Diamond 

Organizational  change  has  been  discussed  in  academic  work  for  the  past  century. 
In  an  effort  to  improve  organizational  efficiency — through  process  improvements, 
structural  changes,  and  new  technology — both  commercial  and  government  sectors  have 
been  avid  consumers  of  newly  developed  approaches  designed  to  resolve  perfonnance 
issues  and  challenges  in  these  areas.  In  order  to  provide  a  theoretical  foundation  for  this 
field  of  study,  organizational  theorists  applied  Systems  Theory  to  model  organizational 
dynamics  and  affects  of  change.  Developed  in  the  19th  century,  Systems  Theory  was 
adapted  to  explore  the  “similarities  in  naturally  occurring  systems  and  human 
organizations.  In  very  basic  terms,  both  take  input  from  their  surrounding  environment, 
subject  it  to  an  internal  transformation  process,  and  produce  some  kind  of  output” 
(Mercer  Delta,  1998,  p.  2).  Feedback  is  then  generated  to  influence  the  input  element  of 
the  system,  as  depicted  in  Figure  2. 
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Figure  2.  The  Basic  Systems  Model  (From  Mercer  Delta,  1998,  p.  3) 


One  of  the  most  prominent  “systems  thinkers”  to  apply  systems  theory  to 
organizations  was  Dr.  Harold.  Leavitt.  According  to  Leavitt,  “one  can  view  industrial 
organizations  as  complex  systems  in  which  at  least  four  interacting  variables  loom 
especially  large;  task  variables,  structural  variables,  technology  variables,  and  human 
variables”  (Leavitt,  1965,  p.  1144).  Leavitt  defined  these  primary  organizational 
variables  as  follows: 

•  Task:  refers  to  industrial  organizations — the  production  of  goods  and 
services,  including  the  large  numbers  of  different,  but  operationally 
meaningful,  sub  tasks  that  may  exist  in  complex  organizations. 

•  Actors:  refers  chiefly  to  people,  but  with  the  qualification  that  acts 
executed  by  people  at  some  time  or  place  need  not  remain  exclusively  in 
the  human  domain. 

•  Technology  (Information  and  Control):  refers  to  direct  problem-solving 
inventions  like  work-measurement  techniques  or  drill  presses. 

•  Structure:  refers  to  systems  of  communication,  systems  of  authority,  and 
systems  of  work.  (Leavitt,  1965,  p.  1 144) 

Leavitt  further  suggested  these  variables  are  highly  interdependent  and  that  a 
change  to  one  will  effect  corresponding  change(s)  in  one  or  more  of  the  other  variables. 
The  resulting  interdependency  is  perhaps  the  most  significant  concept  of  Leavitt’s  work 
in  this  area,  as  it  provides  a  holistic  approach  to  understanding  and  problem  solving  in  an 
organization.  The  interdependency  proved  that  continuous  efforts  to  improve  system 
output  efficiency  through  consideration  of  just  one  variable  are  incomplete  and  often 
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unsuccessful  in  achieving  the  desired  effects  due  to  unintended  effects  on  the  static 
variable(s).  Leavitt  provides  such  an  example: 

The  introduction  of  new  technological  tools — computers,  for  example — 
may  cause  changes  in  structure  (e.g.,  in  the  communication  system  or 
decision  map  of  the  organization),  changes  in  actors  (their  numbers,  skills, 
attitudes,  and  activities),  and  changes  in  perfonnance  or  even  definition  of 
task,  since  some  tasks  may  now  become  feasible  for  the  first  time,  and 
others  become  unnecessary.  (Leavitt,  1965,  p.  1 145) 

Leavitt  also  concluded  that  organizations  must  be  considered  because  they  share 
attributes  of  an  open  system  by  existing  and  being  influenced  by  a  dynamic  environment 
that  can  dramatically  influence  system  variables.  Figure  3  captures  the  Leavitt  Diamond, 
with  the  environmental  consideration  to  provide  the  holistic  view  of  a  complex 
organization  that  provides  the  foundation  for  this  research  (Leavitt,  1978,  p.  286). 


Figure  3.  Leavitt  Diamond  (From  Carroll  &  Sundland,  2008,  p.  25) 

2.  Congruence  Model 

The  work  of  Leavitt  and  other  “systems  thinkers”  provided  the  foundation  for 


subsequent  organizational  models  and  diagnostic  tools.  One  such  application  is  the 
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Figure  4.  Key  Organizational  Components  (From  Mercer  Delta,  1998,  p.  8) 


The  Congruence  Model  suggests  this  deeper  understanding  of  the  entire  system 
can  be  used  to  determine  “fit”  among  the  components.  The  concept  of  “fit”  addresses  the 
alignment  or  congruence  of  the  system  variables,  which  can  then  be  used  to  address 
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overall  system  performance  issues  in  the  holistic  manner  suggested  by  Leavitt.  The 
Congruence  Model  suggests  organizations  must  perfonn  this  comprehensive  self-analysis 
prior  to  considering  major  changes  to  systems  variables  and/or  to  address  changes  in  the 
environment.  The  analysis  must  result  in  an  understanding  of  the  current  and/or 
anticipated  state  of  “fit”  among  system  components,  while  recognizing  that  improved 
congruence  will  improve  overall  system  performance.  Put  another  way,  the  model 
suggests  “the  interaction  between  each  set  of  organizational  components  is  more 
important  than  the  components  themselves  [...]  the  degree  to  which  the  strategy,  work, 
people,  formal  organizations,  and  operating  environment  are  tightly  aligned  will 
determine  the  organization’s  ability  to  compete  and  succeed”  (Mercer  Delta,  1998,  p.  10). 
Figure  5  depicts  the  major  elements  of  the  Congruence  Model. 


Input 

Environment 

Resources  strategy 

History 


Informal 

Organization 


Formal 

Organization 


People 


Output 

System 

Unit 

Individual 


Figure  5. 


The  Congruence  Model  (From  Mercer  Delta,  1998,  p.  14) 


This  research  focuses  on  the  potential  benefit  KM  tools  to  improve  “fit”  among 
acquisition  system  components  as  a  means  to  achieve  improved  system  output  efficiency 


11 


through  implementation  of  policy  objectives  such  as  CPM  and  OA.  The  Leavitt 
Diamond  and  Congruence  Model  are  useful  in  this  context  because  they  stress  the 
interdependency  among  variables/components.  It  is  not  the  purpose  or  intent  of  this 
research  to  analyze  each  variable/component  in  great  detail  or  to  suggest  corresponding 
changes  in  the  people,  organizations,  or  processes.  Instead,  this  research  suggests  that 
application  of  KM  tools  may  form  a  sort  of  “glue”  to  improve  the  fit  among  components. 
Subsequent  change(s)  to  other  variables  (i.e.,  structure  and  process)  will  likely  be 
necessary  due  to  implementation  of  these  technologies.  The  improved  fit  among  system 
components  will  improve  overall  system  perfonnance  and  efficiency  through  increased 
Program  Self-awareness.  The  increase  in  Program  Self-awareness  will  facilitate 
improved  decision  making,  increased  collaboration,  object  re-use,  and  reduced 
development  timelines. 

B.  KNOWLEDGE  MANAGEMENT 

The  information  age  continues  to  shape  the  organizational  environment  and  affect 
all  system  components  of  the  Congruence  Model.  The  fundamental  power  of  personal 
computing,  global  networking,  and  collaborative  technologies  are  essential  to  many 
organizational  processes,  enabling  increased  speed,  availability,  and  volume  of  data  to 
support  decision  making.  These  technology  changes  challenge  organizational  norms  and 
force  organizations  to  perform  self-analysis  to  assess  the  impact  to  the  “fit”  among 
organizational  components  (Mercer  Delta,  1998,  p.  15). 

The  hazards  of  automating  a  bad  process  or  applying  technology  to  outdated 
organizational  structures  are  common  pitfalls  in  the  information  age.  Several 
organizations,  including  Xerox,  recognized  these  hazards  and  applied  the  Congruence 
Model  to  conduct  sweeping  organizational  change.  Xerox  leveraged  technology  to 
achieve  improved  fit  among  components  in  response  to  a  changing  environment — 
resulting  in  a  competitive  edge  in  the  integrated  document  management  marketplace  (p. 
10). 

The  challenges  posed  to  organizations  in  the  information  age  are  many:  turning 
massive  amounts  of  data  into  pertinent  knowledge  and  leveraging  the  potential  of  the 
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network-enabled  “informal  organizations”  to  improve  decision  making.  The  study  of  the 
dynamics  and  potential  of  technology,  process,  and  structure  to  improve  organizational 
knowledge  and  decision  making  has  fueled  both  academic  study  and  technology  research 
and  development  under  the  umbrella  term  of  KM.  The  formal  definitions  of  KM  vary 
among  theorists  and  field  practitioners,  but  they  generally  address  the  common  goal  of 
improving  how  organizations  transform  data  into  knowledge  that  supports  decision 
making.  This  research  focuses  on  how  KM  methodologies  and  tools  can  be  applied  to 
organizations  to  improve  process,  structure,  and  decision  making.  Some  relevant 
definitions  of  KM  include: 

Knowledge  management  is  an  attitude,  not  a  specific  application — a 
commitment  to  taking  full  advantage  of  all  the  information  at  an 
organization’s  disposal  and  delivering  it  to  the  appropriate  constituencies 
to  facilitate  decision  making  at  every  possible  level.  (McKellar,  2009.  p. 

1) 

Knowledge  Management  definitions  span  organizational  behavioral 
science,  collaboration,  content  management,  and  other  technologies  [...]. 
knowledge  and  content  management  technologies  are  used  to  search, 
organize,  and  extract  value  from  all  of  these  information  sources  and  are 
the  focus  of  significant  research  and  development.  These  technologies 
include  text  mining,  clustering,  taxonomy  building,  classification, 
information  extraction,  and  summarization.  (Codey,  2002,  p.  698) 

The  application  of  KM  principles  to  DoD  acquisition  was  the  subject  of  research 
by  military  fellows  at  the  Defense  Systems  Management  College  (DSMC)  in  January 
2000,  titled  “Program  Management  2000:  Know  the  Way.  How  Knowledge  Management 
Can  Improve  DoD  Acquisition”  (Cho,  Hans  &  Landay,  2000).  Figure  6  describes  the 
DSMC  researchers’  concept  of  development  of  knowledge  from  data. 
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The  DSMC  researchers  draw  the  following  conclusions  relevant  to  this  research: 

•  The  commercial  sector  is  successfully  adopting  KM  strategies  to 
achieve  competitive  advantage. 

•  Implementation  of  KM  technologies  in  an  organization  must 
consider  impacts  on  its  people,  processes,  and  structure  to  be 
successful. 

•  KM  initiatives  require  culture  change  and  must  have  the  full 
support  of  the  leadership  to  be  successful. 

•  Mangers  who  effectively  used  their  company’s  knowledge  were 
able  to  overcome  knowledge-based  barriers  and  institutional 
stovepipes  to  improve  collaboration  and  customer  relationships,  as 
described  in  Figure  7. 


14 


Figure  7. 


Knowledge-based  Barriers  and  Solutions  (From  Cho  et  al.,  2000,  p.  2-7) 


•  KM  is  a  source  of  organizational  and  economic  value. 

•  Communities  of  Practice  or  Interest  (COP/COI)  are  forums  of 
networked  people  with  similar  interests  and  issues  that  gather  to 
address  problems,  provide  solutions,  share  ideas,  and  build 
communication  links.  COI  development  provides  the  foundation 
for  KM  implementation. 

•  KM  implementation  should  be  an  incremental  process  built  upon 
on  small  successes.  Figure  8  depicts  the  KM  Framework  as 
continuum  to  capture  this  point. 
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Figure  8.  KM  Framework  (From  Cho  et  al.,  2000,  p.  2-7). 


Cho  et  al.  (2000)  make  a  compelling  case  for  adopting  KM  concepts,  tools  and 
strategy  in  the  DoD  Acquisition  System.  This  research  will  apply  KM  tools  to  specific 
acquisition  problems  that  may  lead  to  the  “small  success”  Cho  et  al.  suggest  is  vital  to 
foster  widespread  KM  adoption  in  DoD  acquisition. 

C.  BUSINESS  INTELLIGENCE  (BI) 

The  KM  field  has  a  close  cousin  in  the  emerging  field  of  Business  Intelligence 
(BI).  BI  captures  a  powerful  set  of  concepts  and  tools  that  are  being  employed  with  great 
success  across  a  range  of  organizations  in  the  commercial  and  government  sectors.  BI 
can  be  defined  as  “an  umbrella  term  that  includes  architectures,  tools,  databases, 
applications  and  methodologies  [...]  to  help  decision  makers  get  valuable  insights  upon 
which  they  can  base  more  informed  and  better  decisions  [...].  The  Process  of  BI  is  based 
on  the  transformation  of  data  to  infonnation,  then  to  decisions,  and  finally  to  actions” 
(Turban,  Shardra,  Aronson  &  King,  2008,  p.  9).  Figure  9  describes  the  many  tools  and 
methodologies  that  comprise  the  BI  field. 
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Figure  9.  Evolution  of  BI  (From  Turban  et  al,  2008,  p.  10) 


BI  draws  upon  the  power  of  computing  and  networking  to  provide  decision 
makers  the  right  information  at  the  right  time  in  an  environment  that  increasingly 
produces  massive  amounts  of  often-uncorrelated  data.  The  following  summary  data  from 
a  survey  of  more  than  500  companies  that  employed  BI  highlights  the  potential  benefits 
relevant  to  DoD  acquisition: 

•  Time  savings  (61%) 

•  Single  version  of  truth  (59%) 

•  Improved  strategies  and  plans  (57%) 

•  Improved  tactical  decisions  (56%) 

•  More  efficient  processes  (55%) 

•  Cost  savings  (37%)  (Turban  et  ah,  2008,  p.  15) 
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D.  KM  AND  BI  TOOLS 

KM  and  BI  share  tools  and  methodologies  that  transform  data  into  information 
and  knowledge,  respectively.  The  tools  relevant  to  this  research  are  detailed  below. 

1.  Data  and  Text  Mining 

DoD  acquisition  programs  generate  massive  amounts  of  documentation  during  all 
phases  of  the  development  process — including  text  documents,  spreadsheets,  structured 
relational  databases,  etc.  The  amount  of  data  and  text  contained  in  these  documents  is 
staggering  but  has  the  potential  for  applying  data  and  text  mining  techniques  to  derive 
useful  information  from  seemingly  unrelated  data. 

Data  mining  is  a  “class  of  information  analysis  based  on  databases  that  looks  for 
hidden  patterns  in  a  collection  of  data,  which  can  be  used  to  predict  future  behavior.  Data 
mining  software  does  not  just  change  the  presentation,  but  actually  discovers  previously 
unknown  relationships  among  the  data”  (Turban  et  ah,  2008,  p.  13). 

Text  mining  is  “the  application  of  data  mining  to  non-structured  or  less  structured 
text  files,  which  entails  the  generation  of  meaningful  numeric  indices  from  the 
unstructured  text  and  then  processing  those  indices  using  various  data  mining  algorithms” 
(p.  224). 

This  research  applies  certain  data  and  text  mining  techniques  to  the  DoD  MDA 
Program  in  an  effort  to  demonstrate  the  potential  for  increased  Program  Self-awareness  to 
support  improved  programmatic  decision  making. 

2.  Data  Warehouses  and  Data  Marts 

Data  mining  techniques  require  that  a  set  of  data  be  defined  such  that  the  various 
data  mining  algorithms  can  be  applied  and  subsequent  analysis  be  perfonned.  This  set  of 
data  is  termed  a  data  warehouse  or  data  mart.  A  data  warehouse  is  a  “physical  repository 
where  relational  data  are  specifically  organized  to  provide  enterprise-wide,  cleansed  data 
in  a  standardized  format”  (Turban  et  ah,  2008,  p.  223).  A  data  mart  can  be  considered  a 
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subset  of  a  data  warehouse,  which  can  be  used  to  support  a  functional  area,  department, 
or  community  of  interest  (p.  222).  These  terms  will  be  used  interchangeably  for  the 
purposes  of  this  research. 

The  development  of  data  warehouses  into  the  structured  form  required  to  support 
data  mining  is  not  a  trivial  process.  The  data  warehouse  needs  to  be  developed  to  support 
the  functional  area  and  include  fundamental  characteristics:  subject  oriented,  integrated, 
time-variant,  and  nonvolatile.  The  data  warehouse  may  also  be  developed  to  include  the 
following  capabilities:  web-based,  relational/multi-dimensional,  client/server,  and 
metadata  (data  about  data)  inclusion  (pp.  39-40). 

Text  mining,  on  the  other  hand,  is  focused  on  developing  new  meanings  and 
relationships  from  unstructured  data  in  the  form  of  documents  (e.g.,  memos,  e-mails, 
instructions,  policies,  etc)  to  support  decision  making.  The  set  of  documents  required  to 
support  text  mining  varies  in  type  and  structure,  providing  more  flexibility  in  formulation 
compared  to  data  warehouse  development.  The  additional  benefit  of  text  mining  is  the 
amount  of  information  available  in  a  fonn  ready  for  processing,  which  includes  upwards 
of  80%  of  the  data  a  typical  organization  collects.  Text  mining  algorithms  are  also 
complex  and  typically  involve  the  following  steps: 

1 .  Eliminate  commonly  used  words  (the,  and,  other); 

2.  Replace  words  with  their  stems  or  roots  (e.g.,  eliminate  plurals,  and  various 
conjugations  and  declarations); 

3.  Consider  synonyms  or  phrases  (e.g.,  student  and  pupil  may  be  grouped); 

4.  Calculate  the  weight  of  the  remaining  terms  (based  on  frequency  of 
occurrence  in  a  document  or  set  of  documents)  (pp.  159-160). 

3.  Analytics  and  Visualization 

The  development  of  data  described  above  supports  its  transfonnation  to 
information  and  knowledge  through  the  process  of  analytics  and  visualization.  Analytics 
includes  a  broad  range  of  capabilities  and  sub-elements  described  in  Figure  10  and  can  be 

defined  as  a  “category  of  applications  and  techniques  for  gathering,  storing,  analyzing, 
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and  providing  access  to  data  to  help  enterprise  users  make  better  business  and  strategic 
decisions”  (Turban  et  ah,  2008,  p.  86).  This  research  will  apply  several  analytical 
applications,  including  data  mining,  text  mining  and  visualization  techniques  to  discover 
relationships  among  program  “features”  to  support  decision  making  regarding  duplication 
of  effort,  gaps,  re-use,  and  collaboration  opportunities  in  the  DoD  MDA  program.  For 
the  purposes  of  this  research,  a  feature  is  a  marketable  behavior  or  property  of  a  system 
ideally  documented  in  a  design — such  as  the  power  window  feature  on  modern 
automobiles. 


Figure  10.  Categories  of  Business  Analytics  (From  Turban  et  ah,  2008,  p.  88) 
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E. 


COLLABORATION 


This  research  has  repeatedly  identified  the  importance  of  collaboration  to  support 
KM  implementation.  The  DSMC  study  heavily  emphasized  the  link  between  KM  success 
and  the  organization’s  culture  of  information  sharing  and  collaboration.  DSMC 
researchers  also  concluded  that  a  typical  DoD  acquisition  program  performs  very  little 
collaboration  across  different  programs  other  than  infonnal  networks  of  functional  area 
associates  fonned  at  the  same  physical  location.  When  development  teams  were  asked 
how  often  they  go  outside  their  program  organization  to  seek  knowledge  to  problems,  the 
most  frequent  response  was  “rarely,  if  ever.”  The  researches  found  it  is  not  that  the  teams 
do  not  recognize  the  potential  power  of  collaboration,  but  they  just  “don’t  know  who  else 
is  working  on  similar  issues  or  don’t  see  any  connection  between  their  project  and 
another  one  in  a  different  area”  (Cho  et  ah,  2000,  p.  1-4).  This  finding  is  not  surprising 
given  the  size  of  the  DoD  acquisition  enterprise,  the  lack  of  enterprise  collaboration  and 
KM  tools,  and  stovepipe  organizational  structures  that  do  not  support  a  culture  of 
information  sharing. 

Despite  these  organizational  and  cultural  challenges,  the  proliferation  of 
networking  technologies  has  penetrated  the  DoD  acquisition  environment.  Several 
collaboration  and  knowledge-sharing  initiatives  have  emerged  in  the  past  decade  that 
may  represent  the  early  stages  of  a  move  towards  greater  collaboration  in  DoD 
acquisition: 

•  FORCEnet  Innovation  &  Research  Enterprise  (FIRE):  Developed  by  the  Naval 
Postgraduate  School  (NPS),  FIRE  is  an  enterprise  information  system  designed  to 
support  Navy  and  Joint  Experimentation.  FIRE  employs  the  latest  web 
collaboration  technologies  to  provide  infonnation  archiving,  document  sharing,  e- 
mail,  and  web  conferencing  capabilities  to  geographically  dispersed 
experimentation  teams  supporting  a  wide  range  of  RDT&E  activities. 

•  DoD  Techipedia:  Developed  by  the  Defense  Technical  Information  Center 
(DTIC),  DoD  Techipedia  is  a  scientific  and  technical  wiki  designed  to  increase 
communication  and  collaboration  among  DoD  scientists,  engineers,  program 
managers  and  operational  warfighters.  This  tool  will  enable  DoD  personnel  to 
collaborate  on  technological  solutions,  reduce  costs,  add  capability  and  avoid 
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duplication.  DoD  Techipedia  will  aid  in  the  rapid  development  of  technology  and 
the  discovery  of  innovative  solutions  to  meet  critical  capability  needs  and  gaps 
(DTIC,  2009). 

•  Software  Hardware  Asset  Reuse  Enterprise  (SHARE)  Repository:  Developed  by 
the  Navy  Program  Executive  Office  of  Integrated  Warfare  Systems  (PEO-IWS) 
and  Naval  Surface  Warfare  Center  (NSWC),  the  SHARE  Repository  serves  as  a 
library  of  ship  combat  system  software  and  related  assets  for  use  by  eligible 
contractors  (both  prime  contractors  and  subcontractors)  for  developing  or 
suggesting  improvements  to  Navy  Surface  Warfare  Systems.  SHARE  fosters 
enterprise  collaboration  to  support  asset  re-use  and  Navy  OA  principles.  (Johnson 
&  Blais,  2008,  p.  1) 

•  AT&L  (Acquisition,  Technology  &  Logistics)  Knowledge  Sharing  System 
(AKSS):  Developed  by  the  Defense  Acquisition  University  (DAU),  AKSS 
provides  acquisition  information  for  all  DoD  service  components  and  across  all 
functional  disciplines.  AKSS  serves  as  the  central  point  of  access  for  all  AT&L 
resources  and  information  and  to  communicate  acquisition  reform.  As  the  primary 
reference  tool  for  the  Defense  AT&L  workforce,  it  provides  a  means  to  link 
together  information  and  reference  assets  from  various  disciplines  into  an 
integrated,  but  decentralized,  information  source.  (DAU,  2009) 


In  recognition  of  the  imperative  and  potential  power  of  collaboration  to  support 
the  complex  DoD  Acquisition  System,  KM  and  acquisition  experts  at  NPS  (Thomas, 
Hocevar  &  Jansen,  2006)  studied  collaboration  in  the  most  complex  DoD  and 
Interagency  acquisitions  to  develop  a  “collaborative  capacity”  assessment  tool.  Figure  1 1 
depicts  the  “Collaborative  Capacity”  model  developed  by  the  NPS  researchers.  The 
notion  that  collective  Self-awareness  is  integral  to  the  success  of  solving  a  common 
problem  can  be  derived  from  this  model.  It  can  also  be  inferred  from  the  model  that 
collaboration  is  the  glue  used  to  bond  stovepiped  organizations  together  to  solve  a 
common  problem. 
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Figure  11.  Collaborative  Capacity  Model  (From  Thomas  et  al.,  2006,  p.  7) 


The  NPS  findings  re-enforce  the  work  of  the  DMSC  fellows  and  highlights  recent 
policy  emphasis  on  collaboration  to  support  implementation  of  best  business  practices. 
The  NPS  research  also  suggests  that  collaboration  in  complex  interagency  acquisition 
programs  is  a  function  of  the  success  and  barrier  factors  described  in  Table  1. 
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Organization 

design 

component 

"Success  "  factors 
that  contribute  to 
collaborative  capacity 

'Barriers  "  that  inhibit 
collaborative  capacity 

Purpose  & 
strategy 

■  'Felt  need'  to 
collaborate 

-  Common  goal  or 
recognized 
intercependence 

-  Adaptable  tc 
interests  ol  other 
organizations 

•  Drvergem  goafs 

-  Focus  on  local  organization 
oser  CTOss-agencr  (eg, 
regional)  concerns 

•  Lack  of  goal  clary 

-  Not  adaptable  to  interests  of 
oner  organizations 

Structure 

Formalized 
coordination 
committee  or  liaison 
roles 

•  Sufficient  aulhonty 
of  paticipancs 

-  Impeding  rules  or  policies 

•  Inadequate  authority  of 
participants 

-  iraoequate  resources 

•  Lack  of  accountability 

Lack  of  formal  roles  or 
procedures  tor  managing 
elaboration 

Lateral 

mechanisms 

-  Social  capital  (i.e., 
interpersonal 
networks) 

•  Effective 

common  <cat>xi  and 
inf  or  nation 
exchange 

-  Technical 
imtercperatuMy 

Lack  of  famikanty  with  otoer 
organizations 

-  Inadequate  comnumcaton 
and  information  staring 
(distrust) 

Incentives 

Collaboration  as  a 
prerequisite  lor 
funding  or  resources 

•  LeaOeislup  suppuit 
and  commitment 

-  Absence  of 

competitive  rvalnes 

-  Acknowledged 
bene'its  of 
collaboration  (e  g  , 

shared  resources) 

-  Competitor  for  resources 

-  Temtonaity 

-  Organization-level  distrust 

-  Lack  of  mutual  lespect 
•  Apathy 

People 

Appreciation  of 
others'  perspectives 

•  Competencies  for 
collaboration 

•  Tiusl 

-  Commitment  and 
motivation 

-  Lack  of  competency 

-  ATogance,  hostilty, 
animosity 

Table  1.  Factors  Affecting  Inter-organizational  Collaboration  (From  Thomas  et  al.,  2006, 

p.  2) 

F.  DOD  ACQUISITION  INITIATIVES 


Two  DoD  acquisition  policy  changes  relevant  to  this  research  are  the  adoption  of 
Open  Architecture  (OA)  approaches  and  Capability  Portfolio  Management  (CPM).  Both 
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OA  and  CPM  are  relatively  young  in  their  implementation  and  address  different  levels  of 
the  acquisition  process,  but  they  share  the  common  goal  of  improving  DoD  decision 
making  regarding  systems-of-systems  (SoS)  acquisitions  to  avoid  duplication,  reduce 
costs,  and  decrease  development  times. 

1.  Open  Architecture 

The  emphasis  on  open  systems  architecture  (OA)  has  increased  over  the  past 
decade,  with  OA  now  being  recognized  as  an  integral  part  of  DoD  systems  engineering 
and  acquisition  processes.  OA  is  not  a  new  concept;  it  draws  from  engineering  design 
principles  that  have  shaped  mature  industries  for  many  decades.  The  modern  automobile 
is  an  example  of  OA  design  principles,  as  it  supports  integration  of  thousands  of  its 
components  through  what  can  be  viewed  as  a  SoS  design.  This  OA  design  allows  most 
components  to  be  built  by  numerous  manufacturers  to  a  standard  interface  specification, 
such  as  tires  built  by  numerous  manufacturers  that  can  fit  onto  the  wheel  of  a  wide  range 
of  vehicles  while  providing  different  levels  of  perfonnance.  The  OA  approach  is  very 
attractive  in  the  context  of  DoD  acquisition,  as  it  offers  potential  for  decreased 
development  timelines  and  reduced  costs  through  re-use  of  components  in  system-of- 
systems  acquisitions.  OA  designs  also  support  quick  upgrades  and  modifications, 
removing  the  requirement  to  redesign  other  components  or  entire  systems.  The 
application  of  OA  to  the  design  of  software-intensive  systems  has  been  the  focus  of  early 
OA  initiatives,  including  the  Navy  PEO-IWS  Software  Hardware  Asset  Reuse  Enterprise 
(SHARE)  Repository  mentioned  above  (Johnson  &  Blais,  2008,  p.  1). 

The  Navy  PEO-IWS  has  provided  the  most  visible  leadership  in  developing  OA 
principles,  concepts,  and  tools  for  the  DoD  acquisition  community.  The  Navy  has  also 
adopted  policies  that  mandate  application  of  OA  design  in  all  SoS  acquisitions  and  that 
define  OA  and  core  principles  as  the  following:  “Naval  Open  Architecture  is  the 
confluence  of  business  and  technical  practices  yielding  modular,  interoperable  systems 
that  adhere  to  open  standards  with  published  interfaces”  (Shannon,  2007,  p.  2). 
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Naval  OA  principles  include: 

•  Building  modular  designs  and  disclosing  data  to  pennit 
evolutionary  designs,  technology  insertion,  competitive  innovation, 
and  alternative  competitive  approaches  from  multiple  qualified 
sources. 

•  Building  interoperable  joint  warfighting  applications  and  ensuring 
secure  information  exchange  using  common  services  (e.g., 
common  time  reference),  common  warfighting  applications  (e.g., 
track  manager)  and  infonnation  assurance  as  intrinsic  design 
elements. 

•  Identifying  or  developing  reusable  application  software  selected 
through  open  competition  of  “best  of  breed”  candidates,  reviewed 
by  subject-matter  expert  peers  and  based  on  data-driven  analysis 
and  experimentation  to  meet  operational  requirements.  (Brummett 
&  Finney,  2008,  p.  20) 

•  Encourage  competition  and  collaboration  through  the  development 
of  alternative  solutions  and  sources.  (Shannon,  2007,  p.  2). 


The  increased  emphasis  on  OA  has  resulted  in  several  initiatives  to  establish 
common  technical  and  architectural  standards  that  will  promote  increased  re-use  and 
interoperability  for  OA  systems,  including  the  SHARE  repository.  These  efforts  are 
critical  to  the  success  of  DoD  OA  implementation  and  require  continued  development  of 
common  vocabularies  and  collaboration  tools.  The  availability  of  such  data  will  facilitate 
users’  and  PMs’  discovery  of  related  efforts  and  potential  re-use  opportunities.  The 
imperative  of  collaboration  in  the  Naval  OA  implementation  is  detailed  in  Figure  12, 
taken  from  a  2007  PEO-IWS  presentation. 


26 
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Figure  12.  Naval  Open  Architecture  Collaboration  (From  Shannon,  2007,  p.  10) 


Another  important  aspect  of  OA  implementation  is  developing  supporting 
information  architectures  with  a  common  vocabulary.  If  the  vocabulary  is  common,  it  can 
describe  similar  system  features  to  enable  acquisition  program  managers  to  correlate 
program  attributes  across  the  range  of  supporting  DoD  RDT&E  and  acquisition  programs 
and  activities.  The  current  process  used  by  Program  Mangers  and  Systems  Engineers  to 
develop  awareness  of  related  RDT&E  efforts  to  identify  potential  re-use  and 
collaboration  opportunities  is  not  well  defined  and  dramatically  limits  the  potential 
advantages  of  OA  acquisitions. 

A  fundamental  requirement  of  OA  is  that  acquisition  managers  develop  an 
awareness  of  related  efforts  and  activities  across  an  enterprise  and/or  community  of 
interest  to  support  decision  making  regarding  duplication  of  effort,  capability  gaps,  re-use 
and  collaboration  opportunities.  Development  of  Program  Self-awareness  is  fundamental 
to  the  success  of  OA  policy  initiatives. 
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2.  Capability  Portfolio  Management  (CPM) 


In  2006,  the  Deputy  Secretary  of  Defense  released  a  memorandum  to  introduce 
the  Capability  Portfolio  Management  (CPM)  approach  to  DoD  Acquisition.  The  intent  of 
exploring  the  CPM  approach  was  to 

manage  groups  of  like  capabilities  across  the  (DoD)  enterprise  to  improve 
interoperability,  minimize  capability  redundancies  and  gaps,  and 
maximize  capabilities  effectiveness.  Joint  capability  portfolios  will  allow 
the  Department  to  shift  to  an  output-focused  model  that  enables  progress 
to  be  measured  from  strategy  to  outcomes.  Delivering  needed  capabilities 
to  the  joint  warfighter  more  rapidly  and  efficiently  is  the  ultimate  criterion 
for  the  success  of  this  effort.  (England,  2006,  p.l) 

The  initial  implementation  of  CPM  included  the  establishment  of  four  capability 
area  test  cases  (i.e.,  Joint  Command  and  Control,  Joint  Net  Centric  Operations, 
Battlespace  Awareness,  Joint  Logistics)  to  evaluate  the  CPM  approach  with  the  long-term 
goal  of  achieving  broader  implementation  in  the  2009-2013  timeframe.  CPM  goals, 
objectives,  and  guidance  emphasized  the  importance  of  system-of-systems  engineering 
approaches  and  “data  transparency”: 

test  case  managers — in  conjunction  with  existing  data  management 
stewards  and  the  Institutional  Reform  and  Governance  effort — should 
work  together  to  establish  an  approach  (business  rules,  data  structure 
changes,  knowledge  management  tools)  that  will  strengthen  the  linkage  of 
authoritative  information  to  capabilities  without  compromising 
information  flexibility.  (England,  2006,  Attachment  A  p.  4) 

CPM  implementation  was  further  directed  across  the  DoD  acquisition  enterprise 
in  2008  and  linked  to  all  nine  Tier  1  Joint  Capability  Areas  (JCA).  The  new  policy 
detailed  CPM  integration  and  alignment  with  existing  DoD  acquisition  structures  and 
processes  to  achieve  widespread  implementation  (England,  2008,  p.l).  The  definition  of 
CPM  was  refined  to  “the  process  of  integrating,  synchronizing,  and  coordinating 
Department  of  Defense  capabilities  needs  with  current  and  planned  DOTMLPF 
investments  within  a  capability  portfolio  to  better  inform  decision  making  and  optimize 
defense  resources.”  (England,  2008,  Glossary,  p.  8) 
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The  CPM  approach  is  relevant  to  this  research  because  it  is  grounded  in 
improving  acquisition  decision  making,  reducing  duplication  of  effort  and  identifying 
capability  gaps.  The  emphasis  on  development  supporting  data  structures,  KM  tools,  and 
implied  expectation  of  expanded  collaboration  provide  a  clear  link  between  DoD  policy 
and  this  research.  KM  tools  directly  support  CPM  decision  making  at  multiple  levels  of 
acquisition — as  will  be  demonstrated  with  the  DoD  MDA  Program — to  identify 
relationships  among  a  portfolio  of  system  features. 
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III.  PROGRAM  SELF-AWARENESS 


A.  PROGRAM  SELF-AWARENESS 

This  research  defines  Program  Self-awareness  as  the  collective  and  integrated 
understanding  of  program  attributes  (i.e.,  system  technology  features,  R&D  activities, 
etc.)  and  their  surrounding  environment  by  program  decision  makers  (i.e.,  program 
managers,  system  engineers,  sponsors).  Program  Self-awareness  is  fundamental  to 
reform  initiatives  such  as  OA  and  CPM,  as  it  enables  decision  makers  to  recognize 
relationships  among  program  attributes  and  seize  collaboration  and  re-use  opportunities 
to  support  cost-effective  acquisitions. 

Achieving  Program  Self-awareness  in  complex  acquisition  programs  such  as  the 
DoD  MDA  program  described  in  Chapter  IV  is  a  lofty  goal  considering  the  myriad 
stakeholders,  processes,  people,  activities,  and  organizational  structures  involved.  This 
research  will  highlight  the  potential  of  KM  tools  to  provide  an  incremental  improvement 
in  Program  Self-awareness.  Figure  13  represents  what  Program  Self-awareness  embodies 
in  the  MDA  COI. 
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Figure  13.  MDA  Program  Self-awareness  (After  Gallup  &  MacKinnon,  2008) 


This  research  suggests  that  DoD  acquisition  decision  makers  could  benefit  from 
applying  KM  tools — such  as  data  and  text  mining,  in  structured  and  unstructured 
program  data  sources  hosted  in  a  COI  data  mart — to  discover  relationships  among 
program  elements  (i.e.,  requirements,  system  features,  activities).  These  previously 
uncorrelated  relationships  could  lead  to  increased  collaboration  within  and  across 
programs  and  improved  COI  Program  Self-awareness  and  integration  of  acquisition 
system  components.  The  Program  Self-awareness  KM  methodology  applied  in  this 
research  to  the  DoD  MDA  COI  is  depicted  in  Figure  14. 
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Figure  14.  Program  Self-awareness  KM  Process 

B.  DOD  ACQUISTION  SYSTEM 

The  DoD  Acquisition  System  is  inherently  complex  due  to  the  processes,  people 
(stakeholders),  and  informal/formal  organizations  that  exist  to  develop,  procure,  and 
sustain  military  capability.  This  research  does  not  seek  to  examine  the  DoD  Acquisition 
System  in  great  detail  to  identify  major  processes,  organization,  or  technology  problems 
that  contribute  to  inefficiencies.  Therefore,  it  is  beyond  the  scope  of  this  research  to 
provide  a  detailed  explanation  of  the  DoD  Acquisition  System  and  surrounding 
environment.  It  is  useful,  however,  to  describe  the  major  components  of  the  DoD 
Acquisition  System  to  demonstrate  the  potential  benefits  of  technology — namely  KM 
tools  and  collaboration — to  improve  the  fit  among  system  components. 

The  primary  processes  that  comprise  the  DoD  Acquisition  organization  and  work 

elements  are  the  Joint  Capabilities  Integration  and  Development  System,  Defense 
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Acquisition  System,  and  the  Planning,  Programming,  Budgeting  Execution  process.  A 
detailed  explanation  of  each  of  the  DoD  Acquisition  Decision  Support  System  elements 
is  provided  at  https://akss.dau.mil/dag/  to  compliment  the  overview  provided  below 
(DoD,  2006,  pp.  1-2). 

•  Joint  Capabilities  Integration  and  Development  System  (JCIDS):  The  systematic 
method  established  by  the  Joint  Chiefs  of  Staff  for  assessing  gaps  in  military  joint 
warfighting  capabilities  and  recommending  solutions  to  resolve  these  gaps.  (pp.  1- 
2) 

JCIDS  is  designed  to  be  event  driven  to  address  emerging  joint  warfighting 
requirements  and  priorities  derived  from  Combatant  Commander  (COCOM)  operational 
needs.  JCIDS  is  also  designed  to  stimulate  Science  and  Technology  (S&T)  and  RDT&E 
in  industry,  government,  and  academia. 

•  Defense  Acquisition  System  (DAS):  The  management  process  by  which  the 
Department  acquires  weapon  systems  and  automated  information  systems. 
Although  the  system  is  based  on  centralized  policies  and  principles,  it  allows  for 
decentralized  and  streamlined  execution  of  acquisition  activities.  (2006,  pp.  1-2) 

The  DAS  is  managed  by  civil  and  military  government  acquisition  officials  who 
comprise  Program  Executive  Offices  (PEO)  and  Program  Management  (PM)  Staffs  and 
Integrated  Product  Teams  (IPTs).  The  DAS  is  designed  to  be  event  driven,  especially  in 
the  system  development  phase,  but  is  greatly  influenced  by  the  calendar-driven  nature  of 
the  PPBE  funding  process. 

•  Planning,  Programming,  Budgeting  and  Execution  (PPBE)  Process:  The 

Department's  strategic  planning,  program  development,  and  resource 

determination  process.  The  PPBE  process  is  used  to  craft  plans  and  programs 

that  satisfy  the  demands  of  the  National  Security  Strategy  within  resource 
constraints  (pp.  1-2). 

The  PPBE  process  is  calendar  driven  to  meet  FY  budget  cycle  timelines  mandated 
by  law  and  is  largely  controlled  by  the  legislative  and  executive  branches  (pp.  1-2). 
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These  three  systems,  as  illustrated  in  Figure  15,  are  designed  to  provide  an 
integrated  approach  to  strategic  planning,  identification  of  needs  for  military  capabilities, 
systems  acquisition,  and  program  and  budget  development  (DoD,  2006,  pp.  1-2). 
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Development 
System  (JCIDS) 

VCJCS/Service 
Chief  Oversight 


Planning,  Programming 
Budgeting  &  Execution 
Process  (PPBE) 

DEPSECDEF 
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Figure  15.  Acquisition  Decision  Support  Systems  (From  DoD,  2006,  p.  2) 


The  decision-support  systems  described  above  are  supported  by  a  complex  array 
of  processes  and  organizations  in  government,  academia,  and  industry.  An  explanation 
of  these  supporting  processes  and  organizations  is  quite  extensive  and  is  available 
through  DoD  acquisition  workforce  training  material  developed  by  the  Defense 
Acquisition  University  hosted  on  the  AT&L  Knowledge  Sharing  System  (AKSS)  website 
(https://akss.dau.mil/default.asp).  For  the  purpose  of  this  research,  the  subordinate 
processes  and  organizations  of  the  decision-support  systems  highlight  another  level  of 
complexity  in  the  DoD  Acquisition  System. 

In  terms  of  system  congruence,  the  fit  among  these  major  system  processes  and 
numerous  supporting  organizations  has  been  of  the  subject  of  a  series  of  GAO  reports. 
The  GAO  has  been  critical  of  the  effectiveness  of  these  systems  in  performing  the  core 
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functions  of  identifying  joint  warfighting  requirements,  controlling  and  forecasting  costs 
and  schedules,  and  reducing  duplication  in  the  portfolio  of  existing  and  planned  systems. 
The  GAO  suggests  the  DoD  “lacks  an  effective,  integrated  approach  to  balance  its 
weapon  system  investments  with  available  resources”  and  that  capability  needs  are  still 
largely  determined  by  the  individual  services  and  programs  that  lack  joint  perspectives, 
resulting  in  duplicative  and  stovepiped  solutions  (GAO,  2008,  September,  pp.  2-3). 

Secretary  Gates  has  been  openly  critical  of  the  effectiveness  of  the  acquisition 
system  in  these  areas,  noting  that  while  “operations  had  become  better  integrated  across 
the  services,  budget  and  procurement  decisions  were  still  largely  separate,  and  sometimes 
duplicative”  (Shalal-Esa,  2009,  p.  1).  In  a  recent  speech  at  the  Marine  War  College, 
Secretary  Gates  asked  the  following  questions  regarding  future  acquisition  reform:  “How 
do  you  move  toward  more  effective  management  of  our  systems?”  “How  do  you  move 
from  joint  operations  to  greater  joint  procurement?”  (McMichael,  2009,  p.  1). 

Given  these  documented  system  inefficiencies  and  the  observed  lack  of 
collaboration  across  acquisition  programs  discussed  in  Chapter  II,  this  research  suggests 
that  poor  fit  among  the  major  components  of  the  DoD  Acquisition  System  is  one  root 
cause  of  system  inefficiencies  that  lead  to  duplication  of  effort  and  limited  re-use  of 
components.  The  consequences  of  these  inefficiencies  and  lack  of  Program  Self- 
awareness  are  increased  acquisition  costs,  delayed  and  lengthy  development  schedules, 
and  reduced  capability  available  to  the  warfighter. 

This  research  suggests  that  DoD  acquisition  decision  makers  could  benefit  from 
application  of  KM  tools  such  as  data  and  text  mining  to  structured  and  unstructured  data 
sources  hosted  in  a  COI  data  mart  to  discover  relationships  among  program  elements 
(i.e.,  requirements,  system  features,  activities).  These  previously  uncorrelated 
relationships  could  lead  to  increased  collaboration  across  programs,  improved  COI 
Program  Self-awareness  and  integration  of  acquisition  system  components.  Figure  16 

applies  the  Congruence  Model  to  the  DoD  Acquisition  System  to  highlight  the  system 
complexity,  area  of  poor  fit  among  components,  and  area  of  opportunity  for  application 

of  KM  tools,  collaboration  and  increased  Program  Self-awareness. 
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Figure  16.  The  Congruence  Model  Applied  to  the  DoD  Acquisition  System  (After 

Mercer  Delta,  1998,  p.  14) 
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IV.  AN  APPLICATION  OF  PROGRAM  SELF-AWARENESS 


A.  MDA  PROGRAM  OVERVIEW 

This  thesis  will  use  the  Maritime  Domain  Awareness  Program  as  a  test  case  to 
qualitatively  analyze  the  utility  of  KM  tools  to  improve  Program  Self-Awareness.  MDA, 
though  having  its  roots  in  traditional  operational  intelligence,  emphasizes  understanding 
the  maritime  environment  in  all  its  facets.  There  has  been  a  litany  of  directives, 
instructions,  and  strategies  written  to  define  and  structure  this  effort.  The  following  is  an 
overview  of  the  program  genesis. 

National  Security  Presidential  Directive  41  (NSPD-41)  and  Homeland  Security 
Presidential  Directive  13  (HSPD-13),  signed  in  December  2004,  define  the  Maritime 
Domain  as  “all  areas  and  things  of,  on,  under,  relating  to,  adjacent  to,  or  bordering  on  a 
sea,  ocean,  or  other  navigable  waterway,  including  all  maritime-related  activities, 
infrastructure,  people,  cargo,  and  vessels  and  other  conveyances.”  NSPD-41  laid  the 
foundation  for  MDA  by  setting  the  following  goal:  “Maximizing  awareness  of  security 
issues  in  the  Maritime  Domain  in  order  to  support  U.S.  forces  and  improve  United  States 
Government  actions  in  response  to  identified  threats.”  NSPD-41/HSPD-13  also 
establishes  a  Maritime  Security  Policy  Coordinating  Committee  (MSPCC)  to  oversee  the 
development  of  a  National  Strategy  for  Maritime  Security  and  eight  supporting 
implementation  plans  (POTUS,  2004,  pp.  2-3). 

The  National  Plan  to  Achieve  MDA,  signed  in  October  2005,  identifies  the  many 
threats  that  face  the  US  within  the  maritime  domain.  The  National  Plan  aims  to 
persistently  monitor,  collect,  fuse,  analyze  and  disseminate,  and  maintain  data  on  vessels 
and  craft,  cargo,  crews  and  passengers,  and  other  identified  areas  of  interest.  These  tasks 
form  the  basis  for  an  “effective  understanding  of  anything  associated  with  the  Maritime 
Domain  that  could  impact  the  security,  safety,  economy,  or  environment  of  the  United 
States  and  [for]  identifying  threats  as  early  and  as  distant  from  our  shores  as  possible” 
(MSPCC,  2005,  pp.  1). 
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The  Secretary  of  the  Navy  (SECNAV)  directed  in  a  May  2007  memorandum  that 
a  prototype  MDA  capability  be  fielded  by  August  2008.  This  document  also  appointed 
the  Deputy  Chief  of  Naval  Operations  for  Communication  Networks  (N6)  and  Deputy 
Under  Secretary  of  the  Navy  (DUSN)  as  co-chairs  of  a  Cross  Functional  Team  (CFT)  to 
oversee  prototype  development.  The  Assistant  Secretary  of  the  Navy  for  Research, 
Development,  &  Acquisition  (ASN  RDA)  appointed  Space  and  Naval  Warfare  Center’s 
(SPAWAR)  Program  Executive  Office  for  Command,  Control,  Communications, 
Computers  and  Intelligence  (PEO  C4I)  as  the  Acquisition  Lead  for  delivery  of  the  Navy 
MDA  Prototype.  The  memorandum  directed  the  fielding  of  an  enduring  operational 
MDA  capability.  This  initial  capability,  called  Spiral- 1,  would  provide  a  technical 
capability  to  the  US  Central  Command  (CENTCOM)  and  US  Pacific  Command 
(PACOM)  Areas  of  Responsibilities  (AORs),  interagency  partners,  and  select  friendly 
and  allied  nations.  Spiral- 1,  at  its  core,  would  create  a  multilayer,  multi-domain  network 
that  would  combine  many  data  streams  into  a  common  operational  picture  (COP) 
accessible  by  US  Government  and  foreign  or  Coalition  partners.  Subsequent  spirals 
would  expand  on  the  capabilities  and  functionalities  of  Spiral- 1  (SECNAV,  2007). 

DoD  Directive  2005.02E,  signed  August  27,  2009,  designated  SECNAV  as  DoD 
Executive  Agent  (EA)  for  MDA.  Under  this  directive,  SECNAV  is  given  authority  over 
the  following: 

•  Oversee  execution  of  MDA  initiatives  within  the  Department  of  Defense  and 
coordinate  on  MDA  policy  with  the  Under  Secretary  of  Defense  for  Policy 
USD(P). 

•  Ensure  continuous,  global,  and  sustainable  support  for  DoD  MDA 
implementation,  coordinated  with  the  DoD  Components. 

•  In  coordination  with  the  Office  of  the  Secretary  of  Defense  Principal  Staff 
Assistant  (OSD  PSA)  and  Under  Secretary  of  Defense  for  Intelligence  USD  (I), 
develop  and  distribute  goals,  objectives,  and  desired  effects  for  MDA  as  they 
pertain  to  DoD  missions. 

•  Identify  and  update  MDA  requirements  and  resources  for  the  effective 
performance  of  DoD  missions,  coordinating  closely  with  the  USD(I)  for  relevant 
intelligence  and  security  matters  and  with  the  Chairman  of  the  Joint  Chiefs  of 
Staff  to  direct  the  consolidation  of  Combatant  Commander  requirements  and  to 
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coordinate  DoD  maritime -related  intelligence,  surveillance,  and  reconnaissance 
initiatives  and  resources  (England,  2008  August  27). 

SECNAV  Instruction  3052.1,  signed  January  2009,  assigns  responsibilities  and 
establishes  the  authorities  and  governance  structure  for  development  and  implementation 
of  comprehensive,  integrated  MDA  activities  for  the  Department  of  the  Navy  (DON) 
(SECNAV,  2009). 

On  March  18,  2009,  the  Chief  of  Naval  Operations  released  a  NAV ADMIN 
message  announcing  the  role  and  objectives  of  the  Navy  MDA  Office.  The  Navy  MDA 
Office  will  report  directly  to  the  Deputy  Chief  of  Naval  Operations  for  Operations,  Plans, 
and  Strategy  (N3/N5).  This  is  a  change  from  the  former  N6  MDA  responsibility.  The 
Office’s  objectives  include: 

•  Ensuring  that  the  right  Navy  stakeholders  are  engaged  in  MDA  development,  that 
the  Navy  presents  a  single,  cogent  MDA  perspective  to  interagency,  international 
and  industry  partners. 

•  Ensuring  that  MDA  capability  and  training  requirements  are  identified  and 
integrated  into  Navy  programming  and  budgeting  processes. 

•  Ensuring  Navy  MDA  efforts  and  investments  are  synchronized  with  the  US  Coast 
guard,  the  Joint  Force  and  other  partners. 

The  Navy  MDA  Office  responsibilities  include: 

•  Providing  the  CNO  an  annual  assessment  of  investment,  engagement,  and 
developmental  efforts. 

•  Coordinating  with  US  Fleet  Forces  Command  (USFF)  to  establish  MDA  priorities 
and  capability  requirements. 

•  Developing  roadmaps  to  align  and  synchronize  Navy  MDA  activities — including 
architecture,  acquisition,  science,  and  technology  (CNO,  2009). 

Through  the  short  history  of  the  MDA  program,  a  top-down  strategy  has 
developed.  A  perceived  need  for  a  fielded  baseline  capability  outweighed  the 
development  of  a  MDA  technical  architecture  into  which  such  component  capabilities 
would  fit,  and  no  set  of  detailed  core  requirements  were  written  to  guide  its  development. 
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There  was  no  clear  direction  on  how  acquisition  organizations  should  cooperate  during 
the  development  process  to  ensure  efforts  were  synchronized  and  emphasized 
interoperability. 

Myriad  organizations  within  the  Global  Maritime  Community  of  Interest 
(GMCOI)  developed  systems  to  provide  this  baseline  MDA  capability.  The  Defense 
Advanced  Research  Project  Agency’s  (DARPA)  Infonnation  Processing  Techniques 
Office  (IPTO)  developed  Predictive  Analysis  for  Naval  Deployment  Activities  (PANDA) 
which  has  subsequently  been  cancelled.  The  U.S.  Naval  Research  Laboratory  (NRL) 
Infonnation  Technology  Research  Group  directs  the  development  of  Comprehensive 
Maritime  Awareness  (CMA)  Joint  Capabilities  Technology  Demonstration  (JCTD)  and 
its  component  application  Track  Assessment  and  ANomaly  Detection — Maritime 
(TAANDEM). 

The  MDA  Program  is  indicative  of  complex  SoS  acquisition  efforts  being 
undertaken  by  the  DoD.  The  MDA  program  includes  additional  complexity  due  to  the 
extensive  international  and  interagency  involvement.  The  program  exhibits  the 
complexities  shown  in  the  Collaborative  Capacity  Model  shown  in  Figure  3.  Figure  17 
gives  the  reader  a  sense  of  the  complexity  of  the  MDA  development  enterprise,  the 
numerous  stakeholders,  and  the  shear  complexity  of  MDA  information-sharing 
relationships. 
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Evolving  Centers  of  Gravity 


Figure  17.  Evolving  MDA  Centers  of  Gravity  (From  Metcalf,  2009) 


It  is  the  goal  of  this  research  to  examine  how  the  numerous  MDA  stakeholders 
might  reduce  duplicative  research  and  development  efforts,  encourage  reuse  of  MDA 
system  components  and  features,  and  facilitate  better  collaboration.  It  will  also 
demonstrate  how  KM  tools  might  help  the  MDA  program  develop  better  Self-Awareness 
of  the  complex  MDA  acquisitions  environment. 

B.  MDA  PROGRAM  SELF-AWARENESS 

MDA  technologies  were  designed  to  provide  the  operator  with  tools  or  features 
that  monitor,  collect,  fuse,  analyze  and  disseminate,  and  maintain  data  on  vessels  and 
craft,  cargo,  crews  and  passengers,  and  other  identified  areas  of  interest.  In  order  for 
decision  makers  to  have  more  visibility  into  what  systems  were  being  developed  by  the 
various  MDA  stakeholders,  they  need  a  high  level  of  Program  Self-awareness. 
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Heightened  Self-awareness  may  lead  to  feature  re-use  through  increased  collaboration  on 
feature  development,  selection  of  the  best  among  like  features,  and  promotion  of  feature 
interoperability. 

This  thesis  proposes  a  notional  KM  methodology  to  support  improved  Program 
Self-awareness  and  decision  making  (Figure  18).  This  research  demonstrates  how  one 
might  go  about  deriving  features  from  existing  program  documentation  and  databases. 
More  importantly,  this  research  will  demonstrate  how  Self-Awareness  can  be  improved 
through  visualizations  of  the  relationships  between  features  of  select  MDA  technologies. 


Figure  18. 


KM  Methodology  to  Support  Improved  MDA  Program  Self-awareness 
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c. 


UNSTRUCTURED  TEXT  MINING  METHODOLOGY 


The  authors  used  the  process  depicted  in  Figure  19  to  conduct  an  analysis  of  an 
MDA  data  set.  This  process  begins  with  the  development  of  business  and  data 
understanding.  The  authors  gained  business  and  data  understanding  by  individual 
research,  conference  attendance,  e-mail  and  phone  discussions,  and  site  visits  to  MDA 
Program  activities.  The  authors  leveraged  NPS  KM  expertise  to  conduct  this  modeling 
and  evaluation.  The  authors  served  as  the  program  “experts.”  The  pairing  of  KM  and 
program  expertise  enabled  the  team  to  understand  the  data  context  and  the  technical 
aspects  of  KM  processes  applied  in  the  research.  The  authors  followed  the  process  in 
Figure  19  to  the  evaluation  step. 


Figure  19.  Data-mining  Process  (From  Turban  et  al.,  2008,  p.  156) 

This  research  develops  and  examines  a  representative  data  mart  of  unstructured 
data  (program  documents)  gathered  from  members  of  the  GMCOI  involved  in  MDA 
systems  development  and  acquisition.  This  task  was  especially  challenging  in  that  there 
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is  not  one  consolidated  repository  for  MDA-related  programmatic  documentation.  The 
data  set  that  the  authors  were  able  to  collect  to  support  this  research  was  not 
comprehensive,  but  represents  a  small  subset  of  MDA-related  documents  available  to  the 
GMCOI. 

Data  collection  was  conducted  via  several  methods.  Access  was  granted  to  the 
PANDA  development  SOURCE  FORGE  collaboration  website  by  the  PANDA  program 
manager  at  DARPA.  PANDA  documents  were  downloaded  into  the  data  mart  from  this 
site,  as  well  as  gathered  directly  from  contacts  involved  in  its  development.  CMA 
documentation  was  gathered  during  a  data  collection  visit  to  SPAWAR  PEO-C4I  1-3  in 
April  2009,  and  directly  from  the  CMA  program  manager  at  NRL.  TAANDEM 
documents  were  gathered  in  the  same  manner. 

The  documents  included  in  the  data  mart  included  Microsoft  PowerPoint  (.ppt), 
Microsoft  Word  (.doc),  Adobe  Acrobat  (.pdf),  e-mails,  .html  fdes,  and  plain  text  fdes 
(.txt).  They  were  grouped  in  to  five  folders  (Figure  20).  Three  of  the  folders  contained 
systems-related  documents  including  TAANDEM,  PANDA,  and  CMA.  Another  folder 
contained  MDA  programmatic  documents.  The  final  folder  contained  MDA-related 
reports  from  the  NPS  Distributed  Information  Systems  Experimentation  (DISE)  research 
group.  The  data  was  subsequently  prepared  for  application  of  the  mining  algorithms. 
Some  scanned  documents  were  converted  with  Adobe  Acrobat  image  recognition 
software  back  into  text.  A  comprehensive  list  of  documents  used  in  this  analysis  can  be 
found  in  the  Appendix. 


Name 

Date  modified 

Type 

,.  CMA 

4/9/2009  10:21  AM 

File  Folder 

DISE 

4/9/200911:18  AM 

File  Folder 

.  MDA  Programmatics 

4/9/2009 11:14  AM 

File  Folder 

PANDA 

4/9/200911:08  AM 

File  Folder 

J  TAANDEM 

4/9/2009 10:24  AM 

File  Folder 

Figure  20. 

MDA  Data  Mart 
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A  data  and  text  mining  toolset  called  Collaborative  Learning  Agent  (CLA), 
developed  by  Quantum  Intelligence,  Inc.,  was  used  to  extract  related  terms  from  the  data 
sets.  CLA’s  association  algorithm  was  used  against  all  5  folders  of  the  MDA  Data  Mart. 
The  tool  divides  plain  text  into  sentences,  grouping  words  from  period  to  period.  The 
tool  eliminates  commonly  used  words  from  the  documents  (i.e.,  a,  the,  and,  other).  The 
tool  then  connects  terms  into  pairs  based  on  their  use  and  proximity  in  the  documents. 
The  result  is  a  list  of  word  pairs  for  each  data  set.  Contacts  and  information  related  to 
Quantum  Intelligence  knowledge  management  tools  can  be  found  at 
http://quantumii.com/. 

These  lists  were  manually  cleansed  for  feature-related  terms.  The  feature-related 
terms  were  chosen  based  on  the  authors’  understanding  of  the  MDA  program.  Words 
such  as  common  names  and  other  terms  unrelated  to  features  were  eliminated  from  the 
word  pair  lists.  The  clean  word  pair  lists  were  then  formatted  for  import  into  select 
visualization  toolsets.  Alone,  these  word  pair  lists  are  probably  of  little  use  in  enhancing 
Program  Self-awareness.  It  is  only  through  the  visualization  of  these  feature  terms  that 
real  relationships  become  apparent  to  the  program  decision  maker. 

After  removing  non-feature -related  terms  from  the  lists,  the  authors  ran  a  script  to 
pair  each  individual  term  with  the  data  mart  folder  it  was  associated  with.  This  allowed 
the  visualization  tools  to  attribute  each  term  to  the  system  or  program  documentation  set 
that  it  came  from.  Figure  20  is  an  excerpt  from  the  formatted  CMA  word  pair  lists. 
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Figure  2 1 .  Excerpt  from  CMA  Word  Pair  List 


The  word  pairs  from  CMA,  TAANDEM,  PANDA,  and  MDA  programmatic  data 
sets  were  then  utilized  in  the  visualization  portion  of  this  research.  After  viewing  the 
terms  extracted,  the  authors  judged  that  the  word  lists  created  from  the  DISE  reports 
would  not  be  as  compelling  to  decision  makers  as  the  other  data  sets.  This  led  them  to 
use  some  other  KM  techniques  on  structured  data  from  the  DISE  FIRE  database 
described  in  Chapter  II. 
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D. 


STRUCTURED  DATA  MINING  METHODOLOGY 


Similar  KM  methodologies  can  be  used  on  structured  data  held  in  databases. 
Many  organizations  use  queries,  reports  and  other  means  to  exploit  known  relationships 
between  data  within  a  database.  KM  techniques  can  be  used  to  identify  previously 
unknown  relationships  between  that  data.  To  demonstrate  some  of  these  capabilities,  the 
authors  utilized  MDA  objective  data  from  the  TRIDENT  WARRIOR  2008  (TW08) 
Experiment  housed  in  the  FIRE  database.  TW08  personnel  collected  data  on  a  number  of 
MDA  toolsets  used  in  the  experiment.  The  objectives  and  results  for  each  tool  set  where 
recorded  in  FIRE.  This  represented  to  the  authors  a  snapshot  of  MDA  capability  from 
the  summer  of  2008. 

The  CLA  association  algorithm  used  with  the  unstructured  data  was  run  against 
the  TW08  data  set.  Word  pairs  were  extracted  from  the  data  set,  and  then  grouped  by  the 
software  based  on  frequency  of  use  within  the  data  set.  Authors  compared  the  lists,  and 
chose  the  list  that  they  most  associated  with  feature  information.  After  the  feature-related 
list  was  chosen,  a  CLA  search  algorithm  was  used  to  find  which  MDA  toolsets  from 
TW08  were  associated  with  each  feature  cluster.  Next,  these  results  were  formatted  for 
visualization. 

E.  VISUALIZATION  METHODOLOGY 

1.  Unstructured  Data  Visualization 

Through  the  use  of  visualization  tools,  such  as  network  analysis  tools, 
relationships  between  the  mined  data  become  useful  to  a  decision  maker.  The  first 
visualization  tool  utilized  in  this  research  was  AutoMap,  developed  by  Carnegie  Mellon 
University’s  Center  for  Computational  Analysis  of  Social  and  Organizational  Systems 
(CASOS)  (Carley,  2005).  Although  AutoMap  can  be  used  to  conduct  text-mining,  it  was 
utilized  only  for  its  visualization  capabilities.  More  detail  on  the  AutoMap  program  can 
be  found  at  http://www.casos.cs.cmu.edu/proiects/automap/. 
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The  first  step  in  this  visualization  methodology  was  to  import  the  word  pair  lists 
into  AutoMap’s  Network  Visualizer.  This  resulted  in  a  display  showing  the  terms  and 
their  relationship  to  other  terms  in  the  imported  network.  The  Organizational  Risk 
Analysis  Network  Visualizer  toolset  connects  terms  that  are  associated  in  the  word  pair 
lists.  For  example,  if  the  word  pairs  (Social  Network)  and  (Network  Analysis)  appear  in 
the  imported  list,  AutoMap  will  fonn  the  relationship  (Social  Network  Analysis).  These 
word  nodes  are  then  connected  by  a  series  of  links  to  form  clusters. 

After  creating  the  visualization,  it  can  be  enhanced  by  grouping  these 
relationships  with  the  Newman  Grouping  tool.  This  tool  displays  each  node  and 
connector  in  a  relationship  by  color.  The  color  clusters  represent  the  degree  to  which  the 
groupings  show  community  structure.  These  clusters  are  based  on  the  statistical 
properties  of  networks  (Carley,  2005).  An  example  of  this  visualization  can  be  seen  in 
Figure  22,  which  depicts  some  of  the  Newman  Groupings  within  the  PANDA  data  set. 

The  reproduction  of  the  visualization  products  in  this  document  is  very  difficult 
due  to  resolution  limits  and  the  inability  to  manipulate  the  data  visualization  through  the 
software  tools  (pan,  zoom,  etc.).  The  graphics  included  do,  however,  provide  a  fair 
representation  of  the  products  this  tool  can  generate. 
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Figure  22. 


Feature  Clusters  from  PANDA  Data  Set  Displayed  in  AutoMap 
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On  the  surface,  Figure  23  is  a  complex  series  of  links  and  nodes,  which  may  not 
be  useful  to  the  casual  observer.  However,  these  clusters  represent  MDA  feature 
concepts  that  have  been  extracted  from  programmatic  documentation.  As  a  program 
decision  maker,  it  would  be  interesting  to  know  what  features  and  capabilities  have  been 
proposed  to  improve  awareness  in  the  maritime  domain.  This  visualization  organizes  the 
feature  concepts  laid  out  in  hundreds  of  pages  of  unstructured  documents.  Even  through 
a  complex  visualization,  an  observer  with  some  knowledge  of  the  MDA  Program  would 
recognize  that  some  of  the  relationships  point  to  specific  capabilities  or  features  desired 
by  the  GMCOI.  Extracting  key  terms  from  existing  documentation  and  displaying  their 
relationships  provides  insight  into  how  the  GMCOI  is  using  the  terms,  and  where  there 
might  be  redundant  terms  and  need  for  de-confliction  and  standardization. 
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Figure  23. 


Feature  Clusters  from  MDA  Programmatic  Data  Set  Displayed  in  AutoMap 
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Figure  24  displays  a  network  map  that  connects  all  the  terms  with  their  pairs  and 
with  the  CMA  system.  This  visualization  will  give  the  viewer  a  sense  of  what  feature 
clusters  the  CMA  system  data  contains  and  how  they  are  related.  This  visualization 
shows  the  feature  clusters  of  a  single  system. 


Figure  24.  Feature  Clusters  from  CMA  Data  Set  Displayed  with  Central  Node  in 

AutoMap 

Decision  makers  may  also  want  to  know  how  two  or  more  systems’  feature 
clusters  compare.  The  following  three  visualizations  (Figures  25,  26  and  27)  were  the 
result  of  concatenating  two  sets  of  word  pairs.  These  visualizations  show  the 
relationships  within  and  between  two  systems.  On  the  outside  edges  of  the  networks  is  a 
large  grouping  of  like-colored  terms  whose  links  radiate  to  the  center  node.  These  terms 
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are  only  connected  to  a  single  system.  The  links  in  the  center,  between  the  two  groupings, 
share  terminology  with  both  systems.  This  is  indicated  by  links  crossing  between  the  two 
circular  concentrations. 
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Figure  25. 


PANDA  and  CM  A  Shared  Feature  Clusters  Visualization 
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Figure  26.  AutoMap  Visualization  of  CMA  and  TAANDEM  Shared  Feature  Clusters 
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Figure  27.  PANDA  and  TAANDEM  Shared  Feature  Cluster  Visualization 
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Additionally,  if  a  user  were  to  concatenate  three  lists,  he  or  she  could  visualize  the 
relationships  among  features  of  all  three  systems  (Figure  28).  There  is  also  a  three- 
dimensional  capability  in  the  AutoMap  visualization  toolset  that  displays  the  three-way 
groupings  much  more  clearly.  These  visualizations  reproduce  poorly  in  document  fonn 
and  were,  therefore,  omitted  from  this  text. 
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Figure  28.  CMA,  PANDA,  and  TAANDEM  Shared  Feature  Cluster  Visualization 
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Deeper  analysis  of  the  feature  clusters  shared  between  systems  can  highlight 
similarities  and  differences  among  the  systems.  An  examination  of  the  shared  hubs 
between  systems  may  give  an  indication  of  these  relationships.  Hubs  are  nodes  with 
multiple  links  extending  from  them.  Figure  29  is  a  closer  view  of  the  PANDA  and 
TAANDEM  feature  clusters.  There  are  several  shared  hubs  between  PANDA  and 
TAANDEM,  including: 

•  port 

•  vessels 

•  posits 

•  deviation 

•  anomaly 

Take  the  “port”  hub,  for  example.  TAANDEM  connects  “port”  with  the  terms 
“regions”  and  “directional.”  PANDA  connects  “port”  with  “common,”  “origination,” 
and  “nearest.”  Another  example  is  the  “vessel”  hub.  TAANDEM  connects  “vessel”  with 
“trackable.”  PANDA  connects  “vessel”  with  “small,”  “suspect,”  “rendezvous,” 
“traversing,”  “flagged.”  The  two  systems  are  both  linked  to  common  hubs  but  display 
different  relationships  with  them.  This  type  of  analysis  might  be  used  to  help 
differentiate  between  similar  features  in  two  different  systems. 


60 


Figure  29.  PANDA  and  TAANDEM  Shared  Feature  Clusters 
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Another  way  to  visualize  which  feature  terms  are  shared  between  systems  is  to 
remove  the  center  node  of  one  system  as  shown  in  Figures  30,  31  and  32.  This  shows 
one  system  in  its  entirety,  and  shows  its  connections  to  feature  terms  from  the  other 
toolset.  This  visualization  makes  it  clearer  within  AutoMap  what  features  are  related  and 
not  related  between  the  systems. 
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Figure  30. 


AutoMap  Visualization  of  PANDA  and  TAANDEM  Feature  Clusters  with  TAANDEM  Node  Removed 
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Figure  31. 


AutoMap  Visualization  of  CMA  and  TAANDEM  Feature  Clusters  with  TAANDEM  Node  Removed 
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Figure  32.  AutoMap  Visualization  of  PANDA  and  CMA  Feature  Clusters  with  CMA  Node  Removed 
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AutoMap  Network  Visualizer  is  a  powerful  tool  for  observing  the  relationships 
between  system-related  feature  terms.  The  display,  however,  is  very  complex,  and  the 
complexity  grows  with  the  number  of  nodes  displayed.  There  are  other  tools  into  which 
these  relationships  can  be  displayed  that  may  have  more  utility  for  the  program  decision¬ 
maker. 

Microsoft  Excel  was  also  used  to  create  visualizations  of  the  relationships 
discovered  through  AutoMap.  Feature  clusters  from  AutoMap  were  exported,  formatted, 
and  imported  to  Excel.  The  clusters  were  sorted  and  displayed  in  radar  graphs  to  answer 
questions  about  data  relationships. 

The  authors  were  interested  in  knowing  where  gaps  existed  in  MDA  capabilities 
given  the  three  systems  being  studied.  The  data  was  sorted  for  feature  clusters  that  only 
appear  in  the  MDA  programmatic  data.  Additionally,  the  authors  were  interested  in  what 
features  were  unique  to  a  single  system.  The  data  was  sorted  for  feature  clusters  that 
were  unique  to  each  system.  Figure  33  displays  MDA  feature  clusters  that  were  not 
associated  with  any  of  the  three  systems  in  red.  It  also  displays  feature  clusters  that  were 
unique  to  a  single  system  represented  by  the  color  in  the  graph  legend.  For  readability, 
some  of  the  feature  clusters  have  been  shortened.  This  type  of  graph  could  serve  as  a  gap 
analysis  for  MDA-related  features.  This  representation  would  be  useful  for  program 
decision  makers  who  want  to  know  what  MDA  capabilities  are  not  addressed  by  existing 
systems  in  the  MDA  Program. 
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Figure  33. 


Radar  Graph  Showing  Gaps  in  MDA  Feature  Coverage,  and  Features  Unique  to  a  Single  System 
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The  authors  were  interested  in  what  feature  clusters  were  shared  by  systems.  In 
order  to  visualize  this,  the  authors  sorted  the  clusters  by  the  number  of  systems  with 
which  they  were  associated.  Figure  34  displays  a  subset  of  feature  clusters  related  to  one, 
two  or  three  systems  represented  by  the  radials  and  rings  of  the  graph.  Once  again,  the 
clusters  are  shortened  for  readability.  The  utility  of  this  graph  in  its  static  format  is 
limited,  but  the  underlying  data  exists  to  determine  which  systems  are  associated  with 
which  capability. 
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Figure  34. 


Radar  Graph  Indicating  Feature  Clusters  Shared  by  One,  Two,  or  Three  Systems 
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?  < 


2. 


Structured  Data  Visualization 


After  the  results  from  the  TW08  were  formatted,  they  were  imported  into  Tree 
Vizualizer.  Tree  Visualizer  is  freeware  available  from  http://mac.softpedia.com/get/ 
Utilities/Tree-Visualizer.shtml.  The  program  displays  interactive  visualization  of 
large  data  structures  organized  in  a  tree.  With  this  tool,  one  is  able  to  visualize  a  data 
structure  quickly  in  its  entirety.  It  also  provides  the  ability  to  quickly  drill  down  to  points 
of  interest  in  a  data  structure. 

Tree  Visualizer  was  used  to  associate  the  feature  clusters  extracted  from  the 
TW08  MDA  objectives  data  set  with  the  MDA  tools  used  in  the  experiment.  Figure  35 
displays  a  hyperbolic  tree  view  of  the  data.  The  MDA  objectives  data  set  is  represented 
by  the  center  node  labeled  MDA2.  The  gray  nodes  surrounding  the  center  node  are 
feature  clusters,  and  the  blue  nodes  on  the  outside  are  MDA  tools  used  in  the  experiment. 
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Figure  35.  Flyperbolic  Tree  Visualization  of  TW08  Objective  Data 
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In  Figure  36,  the  view  is  drilled  down  to  a  specific  feature  cluster.  In  this  case, 
“Distributed”  has  been  clustered  with  “Planners”  “Partners”  and  “Networks.”  Each  of 
these  relationships  is  also  associated  back  to  the  specific  tool  used  in  TW08.  In  this  case, 
feature  terms  from  this  cluster  were  associated  with  4  systems  used  in  TW08,  including 
Google  Earth,  MIDAS,  Global  Trader,  and  CMA. 
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Figure  36.  Flyperbolic  Tree  Graph  of  Feature  Cluster  from  TW08  Objective  Data  Set 
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Another  visualization  that  was  useful  in  displaying  this  data  was  the  icicle  tree 
graph.  In  Figure  37,  the  same  nodes  are  laid  out  in  a  chart.  A  user  can  click  on  a  specific 
term,  in  this  case  “Distributed,”  and  see  the  feature  cluster,  as  well  as  the  tools  associated 
with  the  cluster. 
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Icicle  Tree  Graph  Representing  a  Feature  Cluster  from  TW08 


The  visualizations  in  this  chapter  represent  data  that  were  interesting  to  the 
authors  and  show  potential  utility  for  program  decision  makers.  Other  visualizations 
could  be  produced  depending  on  the  needs  of  the  decision  maker  and/or  program.  These 
visualizations  are  powerful  tools  that  can  be  used  to  improve  Program  Self-awareness  by 
displaying  relationships  among  system  features  within  and  across  programs  that  would 
otherwise  be  very  difficult  or  impossible  to  recognize. 


72 


V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

This  research  has  explored  the  problem  of  duplication,  lack  of  re-use  and 
collaboration  in  DoD  acquisition  and  has  followed  the  intuition  that  increased  Program 
Self-awareness,  enabled  by  KM  tools  and  collaboration,  will  improve  acquisition  process 
efficiencies  in  these  areas.  The  CIFE  research  methodology  depicted  in  Figure  38  was 
used  to  guide  the  research,  as  summarized  in  Table  2. 
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Figure  38.  CIFE  Research  Methodology  (From  Ho,  2007,  p.  2) 
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Observed  Problem 

and  Intuition 

Theoretical  POD  / 
Gap 

Research  Question 

Data  Source 

Duplication  of  effort 
limited  re-use  and 
collaboration  in  DoD 
Acquisition  Programs 
due  to  lack  of  Program 
Self  Awareness 

Systems  Theory  and 
Congruence  Model  —  a 
model  to  understand 
the  acquisition 
environment  and 
Program  Self 

Awareness 

How  can  Knowledge 
Management  methods 
and  tools  be  used  to 
improve  Program  Self 
Awareness, 
collaboration  and  re¬ 
use  in  complex 
acquisition  programs? 

Case  study  of  Navy 

MDA  Program 

-Develop  MDA  Data 

Mart  from  structured 
and  unstructured 
program  data  sources 

Improved  DOD 
Acquisition  Program 
efficiency  and 
effectiveness 

Better  understanding  of 
DoD  Acquisition 

System  knowledge 
environment  and 
potential  for  improved 
Program  Self 

Awareness  enabled 
through  KM  tools 

-  Collaboration  complex 
and  not  efficient 

-  Lack  of  Program  Self 
Awareness  due  to 
complexity  and  culture 
•  KM  tools  can  be 
applied  to  improve 

MDA  Program  Self 
Awareness  and 
decision  making 

-  Mine  Data  Mart  to 
derive  system  “feature" 
data  and  develop 
visualization  tools  to 
show  relationships 
among  system 
attributes 

-  Identify  duplication 
and  opportunities  for 
collaboration,  re-use 
and  efficiency 

Predicted  Impact 

Contribution 

Findings 

Data  Analysis 

Table  2.  Thesis-applied  CIFE  Research  Methodology  (After  Ho,  2007,  p.  2) 


This  research  finds  that  KM  tools  such  as  data  and  text  mining  algorithms,  applied 
to  a  data  mart  of  structured  and  unstructured  programmatic  documents,  can  unearth 
relationships  among  system  features  which  would  otherwise  be  extremely  difficult  or 
impossible  to  recognize.  The  visualization  products  generated  by  these  KM  tools  could 
be  used  by  program  decision  makers  to  identify  duplication  of  effort,  capability  gaps  and 
opportunities  for  component  re-use  and  collaboration  within  and  across  acquisition 
programs  and  RDT&E  activities.  This  research  also  finds  that  these  visualization 
products  could  be  used  by  decision  makers  to  measure  program  progress  towards 
requirements  traceability  and  system  best-of-breed  analysis  in  the  RDT&E  phase  of  the 
acquisition  process. 

This  research  also  concludes  that  the  development  of  Program  Self-awareness  by 
program  and  portfolio  decision  makers  is  fundamental  to  the  successful  implementation 
of  OA  and  CPM  reform  initiatives.  The  KM  tools  and  visualization  products  applied  in 
this  research  have  the  potential  to  improve  Program  Self-awareness  by  highlighting 
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The  data  and  text  mining  models  and  tools  applied  in  this  research  are  extremely 
complex  in  design  and  require  significant  expertise  to  effectively  employ — especially  on 
a  data  mart  of  the  scale  that  would  exist  in  a  large  acquisition  program  or  capability 
portfolio.  This  research  leveraged  KM  expertise  at  NPS  and  a  variety  of  cutting-edge 
mining  algorithms  to  perform  an  iterative  process  of  understanding,  cleansing  and 
analyzing  mining  results  from  a  relatively  small  data  mart  of  structured  and  unstructured 
program  data.  This  process  is  not  trivial  and  requires  a  team  of  KM  experts  and  program 
experts  working  together  to  produce  useful  results.  These  results  are  produces  in  the  form 
of  data  visualizations  that  can  then  be  used  to  support  improved  Program  Self-awareness 
and  decision  making.  That  said,  current  efforts  to  improve  Program  Self-awareness  in 
acquisition  programs  (such  as  the  DoD  MDA  Program)  utilizing  human  analysis  of 
uncorrelated  spreadsheets,  two-dimensional  lists  and  unstructured  documents  are  far 
more  expensive  and  less  effective  at  producing  useful  results  to  support  decision  making. 

This  research  also  finds  that  the  size  and  complexity  of  the  DoD  Acquisition 
System  contributes  significantly  to  the  poor  congruence  or  fit  among  system 
components — which  leads  to  infonnation  stovepipes,  duplication  of  effort,  barriers  to 
collaboration  and  re-use  of  components.  Acquisition  programs  such  as  the  DoD  MDA 
Program  exist  at  an  even  higher  level  of  complexity  due  a  requirement  to  integrate  across 
service,  agency  and  international  acquisition  enterprises.  KM  tools  and  collaboration  can 
be  employed  to  address  symptoms  of  poor  fit  by  providing  a  sort  of  glue  or  patch  to 
improve  Program  Self-awareness,  but  do  not  alleviate  the  need  for  significant  reform  of 
the  DoD  Acquisition  System — to  include  organizations,  processes,  and  people — to 
address  the  root  causes  of  system  inefficiencies. 
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It  is  also  important  to  note  that  this  research  does  not  suggest  that  DoD  acquisition 
decision  makers  are  making  poor  decisions  or  are  not  working  extremely  hard  under  great 
pressure  to  deliver  capabilities  desired  by  the  warfighter  in  a  timely  manner.  The  authors 
recognize  the  complexity  of  the  DoD  Acquisition  System  makes  informed  decision 
making  and  Program  Self-awareness  extremely  difficult  given  the  myriad  stakeholders, 
processes  and  organizations  involved  during  all  phases  of  system  development.  Figure 
39  highlights  the  complexity  of  the  acquisition  decision  making  environment  to  capture 
this  point  with  a  bit  of  humor. 
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Figure  39.  The  Acquisition  Warrior  (From  DAU,  2009) 


This  research  finds  the  DoD  MDA  Program  is  representative  of  large  SoS 
acquisition  programs  in  that  decision  makers  desire  increased  Program  Self-awareness  in 
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order  to  control  acquisition  activities  and  resources.  Program  Self-awareness  is 
extremely  difficult  to  achieve  due,  in  part,  to  the  system  complexity  and  cultural  issues 
addressed  in  Chapters  II  and  III.  The  fact  that  available  KM  technology  is  not  being 
applied  to  support  program  decision  makers  was  also  apparent  during  this  research. 
Decision  makers  are  left  the  overwhelming  task  of  managing  large  programs  and 
portfolios  with  fractured,  incomplete  and  uncorrelated  infonnation  on  which  to  base 
decisions.  The  researchers  found  gathering  programmatic  information  on  the  DoD  MDA 
Program  to  populate  the  data  mart  was  cumbersome  despite  the  cooperation  of  many 
within  the  program.  This  experience  exposed  a  need  for  a  central  data  repository  and/or 
web  portal  to  host  the  latest  and  historical  program  information. 

Collaboration  technologies  such  as  web  portals  are  fundamental  to  many 
commercial  development  programs  and  have  been  credited  with  improving  decision¬ 
making  and  efficiency  in  business  applications — very  similar  problems  to  those  facing 
the  DoD  acquisition  decision  makers.  The  incremental  employment  of  KM  and 
collaboration  technologies  in  an  information-sharing  culture  could  improve  SoS 
acquisition  decision  making  by  developing  improved  Program  Self-awareness  through 
discovery  of  relationships  in  massive  amounts  of  previously  uncorrelated  data.  As 
discussed  in  Chapters  II  and  III,  in  applying  these  technologies,  PMs  must  consider  the 
resultant  impact  to  organizations,  people  and  processes.  In  addition,  the  application  of 
such  technologies  will  require  leadership  and  transformation  to  promote  a  culture  of 
information  sharing  and  collaboration. 

B.  RECOMMENDATIONS  AND  FUTURE  RESEARCH 

Given  the  conclusions  and  findings  above,  it  is  important  to  note  that  the  KM 
methods  and  tools  applied  in  this  research  do  not  provide  the  “magic  bullet”  to  eliminate 
inefficiency  in  the  DoD  Acquisition  System.  These  tools  hold  great  promise  for 
improving  the  quality  of  information  available  to  support  decision  making,  but  require  a 
range  of  actions  (including  those  described  below)  to  preclude  false  starts  and  and/or 
failed  implementation. 
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The  authors  strongly  recommend  a  continuation  of  the  research  of  KM-enabled 
Program  Self-awareness.  This  follow-on  research  could  refine  the  visualization  products 
applied  in  this  research  and  explore  other  applications  which  would  enable  user- 
controlled  drill-down  to  further  expose  information  useful  to  decision  makers.  This 
follow-on  research  could  also  develop  measures  of  program  performance  by  providing 
correlation  and/or  traceability  of  system  features  to  requirements;  such  research  could 
also  expose  the  potential  for  these  tools  to  facilitate  definition  of  system  features  and 
common  vocabularies  based  on  semantic  relationships  generated  during  the  KM 
processes. 

It  is  the  recommendation  of  the  researchers  that  a  program  similar  to  the  DoD 
MDA  Program  be  selected  as  a  test  case  to  apply  the  KM  methods  and  tools  described  in 
this  research.  The  researchers  believe  this  test  case  would  generate  very  useful  results 
and  provide  an  incremental  success  and  learning  curve  to  streamline  subsequent 
application  to  other  acquisition  programs. 

The  DoD  should  invest  in  KM  and  collaboration  technologies  and  move  beyond 
the  current  acquisition  information  environment — which  is  comprised  largely  of 
information  silos  contained  in  e-mails  chains,  phone  and  video  conferences,  and  other 
isolated  data  sources  of  program  information.  Acquisition  programs  and  related  COIs 
should  develop  web  portals  to  serve  as  information  repositories  of  the  latest  program 
information — such  as  requirements  documents,  development  updates,  and  test  and 
experimentation  data.  These  program  documents  could  feed  the  program  data  mart  for 
application  of  the  KM  tools  described  in  this  research.  The  program  portal  could  also 
host  program-related  blogs  and  information  wikis  to  promote  Program  Self-awareness 
and  collaboration  within  and  across  related  programs  and  activities.  These  web  portals 
can  provide  for  controlled  access,  but  must  not  be  overly  restrictive  or  inaccessible  if  they 
are  to  accommodate  unanticipated  membership  and  participation  across  programs  and/or 
related  COIs. 

This  research  has  also  highlighted  recent  frustration  with  the  DoD  Acquisition 
System  at  the  highest  levels  of  government.  There  is  clearly  an  active  search  for 

sweeping  reform  solutions  to  address  DoD  Acquisition  System  inefficiencies  and  their 
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resultant  poor  outcomes.  The  authors  recommend  these  reform  efforts  go  beyond  the 
historical  approaches  of  adding  layers  of  additional  executive  oversight  and  processes 
through  program  reviews  and  decisions  gates.  The  researchers  believe  reform  efforts 
should  devote  significant  time,  energy,  and  resources  to  develop  KM  and  collaboration 
technologies  such  as  those  described  in  this  research.  These  investments  could  improve 
Program  Self-awareness  and  promote  a  culture  of  infonnation  sharing  and  collaboration 
in  the  DoD  Acquisition  System.  As  Secretary  Gates  and  Undersecretary  Young  have 
recently  pointed  out,  the  DoD  Acquisition  System  must  do  better  at  sharing 
information — much  as  the  Services  are  achieving  battlefield  success  through  Joint  and 
net-centric  interoperability.  Perhaps  the  reform  efforts  could  include  a  vision  towards 
“net-centric  acquisition”  and  culture  that  leverages  and  applies  the  principles  of  net- 
centric  warfare  to  the  DoD  Acquisition  System. 

This  research  has  detailed  the  potential  utility  of  KM  tools  and  collaboration 
applied  to  the  DoD  Acquisition  System.  The  impact  of  these  technologies  on  other 
elements  of  the  Defense  Acquisition  System  has  also  been  discussed,  to  include  the  need 
for  a  holistic  system  analysis  towards  achieving  system  congruence.  This  research 
concurs  with  the  conclusions  and  recommendations  of  the  DSMC  researchers  discussed 
in  Chapter  II,  especially  their  call  for  incremental  KM  implementation  approaches  built 
on  small  successes,  development  of  a  supporting  KM  implementation  strategy,  and 
sustained  executive  leadership  throughout  implementation  of  the  process  described  in 
Figure  8.  This  research  further  recommends  the  following  KM  implementation  success 
factors  from  the  commercial  sector  be  considered  in  developing  a  KM  implementation 
strategy  for  the  DoD  Acquisition  System  (Weir,  2002): 

•  The  project  must  fit  with  corporate  strategy  and  business 
objectives. 

•  There  must  be  complete  buy-in  to  the  project  by  executives, 
managers,  and  users. 

•  It  is  important  to  manage  user  expectations  about  the  completed 
project. 

•  The  data  warehouse  must  be  built  incrementally. 

•  The  PM  must  build  in  adaptability. 
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•  The  project  must  be  managed  by  both  IT  and  business 
professionals. 

•  The  PM  must  develop  a  business/supplier  relationship. 

•  Users  of  KM  programs  should  only  load  data  that  have  been 
cleansed  and  are  of  a  quality  understood  by  the  organization. 

•  PMs  should  not  overlook  training  requirements. 
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APPENDIX. 


MDA  DATA  MART  DOCUMENTS 


A.  DISE-RELATED  DOCUMENTS 

(Note:  Though  the  DISE  folder  is  mentioned  in  this  text,  no  data  from  this  folder  was 

actually  used  for  this  research) 

B.  CMA-RELATED  DOCUMENTS 

Boraz,  S.  (2007).  Comprehensive  maritime  awareness  joint  capabilities  technology 
demonstration.  Naval  Research  Lab  2007  Review.  Washington,  DC:  Author. 

Naval  Research  Lab  (NRL).  (2009,  March  6).  Comprehensive  maritime  awareness 
(CMA)  joint  capability  technology  demonstration  (JCTD)  functional 
familiarization  manual  (FFM)  (SSD-D-CMA215  Revision  J  OD4  Build  2). 
Washington,  DC:  Author. 
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Moore,  K.  (2009,  February  9).  Program  Manager  from  DARPA.  [E-mail  correspondence 
with  researcher.]. 
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Unpublished  PowerPoint  Presentation.  Washington  DC. 
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Presentation.  Washington,  DC. 
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Systems  Engineering  and  Integration  (SE&I)  Lockheed  Martin  Advanced  Technology 

Laboratories.  (2008,  March  10).  System  architecture  for  the  predictive  analysis 
for  naval  deployment  activities  (PANDA)  Phase-II  system  revision  1.0. 
Unpublished  program  document.  Cherry  Hill,  NJ. 

DARPA  IPTO.  (2008,  April  17).  Concept  of  operations  (CONOPs)  for  predictive 
analysis  for  naval  deploymen  t  activities  (PANDA).  (Ver.  1.0).  Unpublished 
program  document.  Washington,  DC. 

DARPA  IPTO.  (2008,  March  14).  Framework  proposal  draft  design  document. 
Unpublished  program  document.  Washington,  DC. 
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researchers.]. 
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